Google Cloud Datalab provides a set of commands for working with data stored in Google Cloud Storage. They can help you work with data files containing data that is not stored in BigQuery or manage data imported into or exported from BigQuery.
This notebook introduces several Cloud Storage commands that Datalab introduces into the notebook environment.
In [4]:
%%gcs --help
First, a couple of commands to list Datalab sample data. Try %%gcs list without arguments to list all buckets within the current project:
In [ ]:
%%gcs list
In [5]:
%%gcs list --objects gs://cloud-datalab-samples
Out[5]:
You can also use wildchars to list all objects matching a pattern:
In [9]:
%%gcs list --objects gs://cloud-datalab-samples/udf*
Out[9]:
In [14]:
# Some code to determine a unique bucket name for the purposes of the sample
from google.datalab import Context
import random, string
project = Context.default().project_id
suffix = ''.join(random.choice(string.lowercase) for _ in range(5))
sample_bucket_name = project + '-datalab-samples-' + suffix
sample_bucket_path = 'gs://' + sample_bucket_name
sample_bucket_object = sample_bucket_path + '/Hello.txt'
print('Bucket: ' + sample_bucket_path)
print('Object: ' + sample_bucket_object)
NOTE: In the examples below, the variables are referenced in the command using $ syntax since the names are determined based on the current project. In your scenarios, you may be able to use literal values if they are constant instead of creating and using variables.
In [15]:
%%gcs create --bucket $sample_bucket_path
In [16]:
%%gcs list --objects $sample_bucket_path
Out[16]:
In [17]:
%%gcs copy --source gs://cloud-datalab-samples/hello.txt --destination $sample_bucket_object
In [18]:
%%gcs list --objects $sample_bucket_path
Out[18]:
In [19]:
%%gcs view --object $sample_bucket_object
Out[19]:
In [20]:
%%gcs read --object $sample_bucket_object --variable text
In [22]:
print(text)
In [23]:
text = 'Hello World!\n====\n'
In [24]:
%%gcs write --variable text --object $sample_bucket_object
In [25]:
%%gcs list --objects $sample_bucket_path
Out[25]:
In [26]:
%%gcs delete --object $sample_bucket_object
In [27]:
%%gcs delete --bucket $sample_bucket_path